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Summary 


States and districts across the country are adopting new strategies to evaluate teacher effec- 
tiveness. These strategies are intended to assess educators’ strengths and limitations and 
guide decisions about opportunities for professional growth. Yet little research has explored 
how teacher evaluation data are used in administrative decisions. This study describes how 
data from teacher evaluations are used in five volunteering Arizona school districts, with 
a focus on how the data guide teachers’ professional development and influence decisions 
about teachers’ compensation, assignment to classrooms or schools, and remediation and 
retention. 

During the 2014/15 school year the study districts administered their own teacher evalu- 
ation systems, which were developed to align with state evaluation regulations passed in 
2011. Because of the variations in system design and implementation, this study treats each 
district as an independent case. The study team analyzed interviews with district officials 
and instructional coaches and online surveys of school principals and teachers, collect- 
ed by West Comprehensive Center on behalf of the Arizona Department of Education, 
and produced five district case summaries, which are included in appendix C. The report 
summarizes common experiences, practices, and perceptions related to the use of teacher 
evaluation data reported across three or more of the districts: 

• Online systems facilitate timely observation-based feedback. District officials in the 
four districts with an online system that supports their observational framework 
(either developed by the district or supported by a vendor) emphasized that the 
system affords teachers access to considerable feedback in a timely manner and 
promotes transparency and common understanding when teachers meet face-to- 
face with their coach or evaluator. 

• Evaluation data influence subsequent professional development. Evaluation data 
shape the work of instructional coaches (who play no formal role in evaluations) 
and the support opportunities (books, webinars, and online videos) that are sug- 
gested for teachers within district online systems. In three districts more than 
half of responding teachers agreed that they engage in professional development 
opportunities linked directly to the needs identified in their evaluations, and in 
four districts more than half of responding teachers agreed that the next steps they 
need to take for their professional growth are clear to them after their evaluations. 

• Student test data are incorporated into teachers’ final performance classifications in 
multiple ways, but observation data are perceived as more useful for professional devel- 
opment decisions because they are collected over repeated occasions and made avail- 
able during the school year. In accordance with Arizona requirements the districts 
incorporate student test data into teachers’ final performance classifications in 
various ways. Elowever, district officials reported that the summative nature of 
these teacher-level data (which are usually conveyed as a single score) limit their 
formative value. When results are based on statewide exams, the data are not 
released until summer and thus are not reviewed with teachers until the beginning 
of the next school year — which is also reportedly problematic. 

• Teachers view themselves as responsible for their own professional growth and are 
somewhat skeptical of school and district professional development. 

• Evaluation data are not systematically used to identify teacher leaders or to assign 
teachers to schools or classrooms, but such data serve as the basis for decisions on reme- 
diation and allocation of state performance pay funds. 


• Teachers were more skeptical than administrators about the benefits of their new eval- 
uations. District officials acknowledged anxiety among teachers about the evalua- 
tion process implemented in response to the 2011 state requirements, particularly 
related to component measures such as test scores and student and parent surveys 
that teachers suspect may be unreliable and unduly influenced by factors beyond 
their control. 

Findings from the study suggest positive benefits from the organizational structures that 
support the review of evaluation data during the school year — standards-based observation 
frameworks, benchmark assessments, professional learning communities, and instructional 
coaching and feedback. The evident skepticism suggests that teachers may not perceive 
evaluations as entirely credible and relevant to their work. 

The study did not attempt to characterize the fidelity or quality of the implementation of 
local teacher evaluations. The study is limited by its small, voluntary sample of districts as 
well as by low survey response rates among teachers. 
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Why this study? 


States and districts across the country are adopting new strategies to evaluate teacher 
effectiveness. Because effective teaching is consistently linked to improved learning out- 
comes for students, states are eager to see districts implement systems in a timely manner 
that provide schools with research-supported tools and methods for evaluating their staff 
(Darling-Hammond, 2010; Rivkin, Hanushek, & Kain, 2005; Watson, Kraemer, & Thorn, 
2009). In some states, evaluation data provide feedback that teachers can use formatively 
to improve their instruction. The data may also be used for higher stakes purposes, such as 
decisions about contract renewal (Jerald, 2012). 

Many state regulations of teacher evaluation are based on the premise that instructional 
feedback and support to teachers will become more targeted and relevant once districts 
implement new multiple-measure evaluation systems. Early results from a 2012/13 initial 
implementation of the Arizona Department of Education’s model evaluation system have 
been promising. The department developed its own teacher evaluation model aligned with 
state requirements, which districts can adopt, that includes observations of instructional 
practice in classrooms using the Danielson Framework for Teaching; online surveys of stu- 
dents, parents, and peer teachers; and measures of student academic progress, defined by 
state-created rating tables based on teaching assignment (Arizona Department of Educa- 
tion, 2013). Participating teachers in the four 2012/13 pilot districts, two of which also par- 
ticipated in the current study, reported that post-observation conferences provided them 
with meaningful feedback on how to improve their instruction and that they were more 
reflective about their instructional practice and professional growth in 2012/13 than in 
previous years (Ruffini, Makkonen, Tejwani, & Diaz, 2014). 

However, little is known about the next steps that these teachers took to improve their 
practice through professional development. Simply administering measures and making 
results available do not ensure that evaluation data will be used in professional develop- 
ment decisions (Marsh, 2012). Even states that provide substantial direction to district 
officials about how to collect and use evaluation data are finding that local practices vary 
widely (National Council on Teacher Quality, 2011). Traditionally, principals have best 
promoted their teachers’ development by playing a facilitative role (Crum & Sherman, 
2008; Seashore Louis, Leithwood, Wahlstrom, & Anderson, 2010) through, for example, 
providing actionable feedback to teachers (Hattie, 2009), developing communities of prac- 
tice where teachers can collaborate (Wahlstrom & Louis, 2008), or creating professional 
development systems in which teachers have the opportunity to routinely develop and 
refine their skills (Bryk, Sebring, Allensworth, Luppescu, & Easton, 2010). States and dis- 
tricts are today seeking guidance about how the data from expanded evaluations will be 
beneficial to educators and students (Doherty & Jacobs, 2013). 

A collaboration between Regional Educational Laboratory West’s Educator Effectiveness 
Alliance and the Arizona Department of Education, this study aims to help state offi- 
cials understand how the districts use teacher evaluation data to guide teachers’ profes- 
sional development and influence decisions about teachers’ compensation, assignment to 
classrooms or schools, and remediation and retention. The study follows previous research 
on the initial implementation of teacher evaluation systems aligned with the statewide 
Arizona Framework (Ruffini et al., 2014), which examined practitioners’ perceptions of the 
new measures and processes rather than the use of evaluation data. 
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What the study examined 


This report describes how data from teacher evaluations are used in five Arizona school 
districts and summarizes common findings across the study districts. Each district admin- 
istered its own teacher evaluation system developed to align with state evaluation regula- 
tions passed in 2011 (the Arizona Framework for Measuring Educator Effectiveness; box 1). 

The study addressed two questions related to the five sample school districts: 

• How were data from the teacher evaluation process shared and used by district 
administrators, principals, and teachers? 

• What were the similarities among the study districts in their use of teacher evalu- 
ation data? 

See box 2 for a summary of the study methodology and appendix A for more details. 


Box 1. The Arizona Framework for Measuring Educator Effectiveness 

In 2010 Arizona Senate Bill 1040 mandated that local education agencies annually evaluate 
teachers and use the data to improve teaching and learning for all students. Subsequently, in 
April 2011 the Arizona State Board of Education approved the Arizona Framework for Measur- 
ing Educator Effectiveness (Arizona State Board of Education, 2013), guidelines requiring that 
teacher evaluations include data from measures of teaching performance (for 50-67 percent 
of each teacher’s final performance classification) and of student academic progress (at the 
classroom level and, optionally, at the school level, for 33-50 percent). 

Some Arizona teachers (known as group A teachers in the 2011 Arizona Framework) have 
access to classroom-level student achievement data from measures that are valid and reli- 
able, aligned to state academic standards, and appropriate for the content area (for example, 
a standardized state assessment); other teachers (known as group B teachers in the 2011 
Arizona Framework) do not have access to comparable classroom-level data (Arizona State 
Board of Education, 2013). 

Within the two component areas — teaching performance and student academic progress 
— the Arizona Framework affords local flexibility, allowing districts to identify their own mea- 
sures and combine and weight them using strategies that best fit their culture and context. In 
summer 2011 school districts and charter schools in Arizona began designing and implement- 
ing their own Arizona Framework-aligned teacher evaluation systems. As noted, the Arizona 
Department of Education also developed and piloted its own Arizona Framework-aligned model 
that districts could adopt (see http://www.azed.gov/teacherprincipal-evaluation/). 

The Arizona Framework recommends that the teacher evaluation process, and the data from 
it, be used “to drive professional development to enhance teaching” as well as to “create a 
culture where data drives instructional decisions” (Arizona State Board of Education, 2013, p. i). 

Evaluation data also are intended to have professional consequences, in that districts 
must use evaluation data to classify teachers into one of four overall performance classifica- 
tions (highly effective, effective, developing, or ineffective). State law (Arizona Revised Statutes 
§15-537) requires districts to provide incentives for highly effective teachers, while teachers 
rated ineffective for two consecutive years “shall not be transferred as a teacher to another 
school" and must complete “a professional development program focused on the areas in 
which the teacher needs to improve." Moreover, teachers beginning their fourth year of employ- 
ment who are designated as ineffective may be offered noncontinuing status contracts. 
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Box 2. Methodology 


The Arizona Department of Education adopted a purposeful stratified sampling approach to 
invite districts to participate in the study in October 2014, initially targeting 17 districts that 
varied in enrollment size, student demographics, standardized test performance, and teacher 
effectiveness measures implemented. Among the 17 districts invited, districts A-E volunteered 
to participate. Districts C and D had collaborated with the Arizona Department of Education to 
pilot its state model teacher evaluation process in the 2012/13 and 2013/14 school years. 

The 2013/14 enrollment in the five districts ranged from fewer than 1,000 to more 
than 25,000 students, and the percentage of racial/ethnic minority students ranged from 
33 percent to 91 percent. Under the state’s A-F letter grade accountability system, based on 
spring 2014 standardized test scores, two of the districts were awarded the letter grade A, 
two the letter grade B, and one the letter grade C. During the study period each district admin- 
istered its own teacher evaluation system aligned with the Arizona Framework for Measuring 
Educator Effectiveness. 

The study team analyzed data from structured interviews with 19 district officials and 
instructional coaches, as well as from online surveys from 47 principals (76 percent response 
rate) and 971 teachers (35 percent response rate), conducted by West Comprehensive Center 
on behalf of the Arizona Department of Education, between January 2015 and May 2015 (see 
table A1 in appendix A for the number and role of the officials interviewed in each district, 
table A2 in appendix A for survey counts and response rates by district, and appendix B for the 
interview protocol). 

Given the local variations in system design and implementation, the study team treated 
each district as an independent case and produced five district case summaries (see appendix 
C). This report summarizes common findings across the districts. Cross-case themes were 
included as findings if the theme was raised in three or more of the study districts. 


Arizona officials can use the results of this study to guide decisions about what types of 
guidance and support the state might provide to districts. Common practices that are per- 
ceived as beneficial by several participating districts may also be of interest to other dis- 
tricts striving to implement similar evaluation reforms. 

What the study found 


This section summarizes common experiences, practices, and perceptions related to the 
use of teacher evaluation data reported by three or more of the study districts. 

Teachers and principals know what data are being collected as part of teacher evaluations and 
interact multiple times during the year to review and discuss standards-based observation data 

Every responding principal and at least 73 percent of responding teachers in each dis- 
trict reported knowing what types of information are collected during the teacher eval- 
uation process. In addition, at least 58 percent of responding teachers in four districts 
(all except district C) reported knowing how the collected data would be used by school 
administrators. 


Every responding 
principal and at 
least 73 percent 
of responding 
teachers in 
each district 
reported knowing 
what types of 
information are 
collected during 
the teacher 
evaluation process 


As part of teacher evaluations all five districts rely on standards-based observation frame- 
works, which are carried out through multiple formal and informal observations over the 
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course of the year. District officials in all five districts also reported that their observation 
frameworks provide a common language for teachers, instructional coaches, and principals 
to discuss instructional practice. 

According to district officials from all five districts, at a minimum, principals have formal 
meetings with teachers at the beginning and end of the school year to review evaluation 
data and discuss professional goals. While instructional coaches do not formally contribute 
to evaluations in any district, they observe teachers and review student benchmark assess- 
ment data with them, offering feedback that may be used formatively over the course of 
the year. 


Finally, officials from all five districts emphasized the importance of monitoring inter-rater 
reliability and reported that their principals are trained to understand the observation 
rubric and are expected to rate teaching practices consistently across observations. 

Online systems facilitate timely observation-based feedback 

District officials cited benefits from having an online system, either developed by the dis- 
trict or supported by a vendor, that is aligned with their observational framework and used 
to collect and share observation data with teachers. School administrators or coaches can 
enter data and notes into such systems during or after formal or informal observations, 
and teachers can immediately access and review data and enter comments of their own. 
According to district officials in the four districts with online data systems (A, B, D, and 
E), such systems afford teachers timely access to feedback and promote transparency and 
common understanding when teachers meet face-to-face with their evaluators and coaches. 


School and 
district officials 
in all five districts 
reported that 
teacher evaluation 
data influence 
the professional 
development 
opportunities that 
are subsequently 
offered to teachers 


Evaluation data influence subsequent professional development 

School and district officials in all five districts reported that teacher evaluation data influ- 
ence the professional development opportunities that are subsequently offered to teachers. 
Officials attributed this connection to their history with their standards-based instruction- 
al framework and observational rubric. For districts, A, C, and D this was the Danielson 
Framework for Teaching, for district B it was the Strange Teacher Effectiveness Perfor- 
mance Evaluation System, and for district E it was the Marzano Teacher Evaluation Model. 
In each district these tools are used to define effective teaching across multiple domains 
and to identify teachers’ strengths and weaknesses. For example, the district D superinten- 
dent explained that after reviewing teachers’ effectiveness in particular areas, the district 
identified domain 2 (classroom environment) of the Danielson Framework for Teaching 
as an overarching area of need — one that emerged as a challenge for many teachers — and 
dedicated districtwide trainings toward improving performance in that area. 

Teacher survey data reinforced a connection between evaluation and professional devel- 
opment. More than half the responding teachers in three districts (A, D, and E) agreed 
that they engage in professional development opportunities that are linked directly to the 
needs identified in their evaluations, and more than half the responding teachers in four 
districts (all except district C) agreed that the next steps they need to take for their profes- 
sional growth were clear to them after their evaluations. 
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District officials in the four districts with online data systems (A, B, D, and E) said that their 
system provides suggested resources or support opportunities (such as books, webinars, and 
online videos) linked to particular domains or components within the observational frame- 
work. Officials in all five districts also emphasized that instructional coaches play a key role 
in guiding teachers’ professional growth by working with teachers (either individually or in 
teams) to help them improve their practices (based partly on prior evaluation data from prin- 
cipals). This support takes various forms, such as helping teachers develop their professional 
growth plans or analyze their students’ benchmark assessment data or conducting informal 
follow-up observations to track progress and offer feedback. In addition, in districts B and D 
instructional coaches help principals design their schoolwide professional development plans. 

Student test data are incorporated into teachers’ final performance classifications in multiple ways, 
but observation data are perceived as more useful for professional development decisions because 
they are collected over repeated occasions and made available during the school year 

Districts incorporate student test data into teacher evaluations in various ways. For 
example, districts A and B apply a value-added model 1 to statewide student test data and 
share the resulting teacher scores with principals during the summer; the principals in 
turn discuss the scores with teachers in the fall. Both districts have safeguards to shield 
teachers from invalid inferences due to unreliability or scoring error. 2 In districts C and D, 
which have adopted the Arizona Department of Education’s model, evaluators rely on state 
rating tables (each with its own student data combinations and formulas) that are specific 
to particular teaching assignments. The tables require teachers to develop student learning 
objectives (test growth targets for their students) that are evaluated by their administrators 
at the end of the year or in the summer. District E bases the student academic progress 
portion of teachers’ performance classifications on the aggregate academic growth of stu- 
dents across their entire school. 

These processes are all aligned with requirements of the Arizona Framework for Measuring 
Educator Effectiveness and allow the districts to incorporate a measure of student academ- 
ic progress into teachers’ final performance classifications. However, the summative nature 
of the results (which are usually conveyed as a single score) can limit their formative value 
in shaping teachers’ professional development. When results are based on statewide exams, 
the data are not released until summer and thus are not reviewed with teachers until the 
beginning of the next school year — which is also reportedly problematic. In contrast, data 
from formal and informal observations can be tracked and shared online and discussed at 
various points during the school year. 

Officials in all five districts emphasized that teachers are expected to review and respond 
to their students’ benchmark assessment data over the course of the year — on their own, 
with instructional coaches, or with peer teachers (table 1). Although such efforts may help 
student test data shape teacher practice in the study districts, the utility of teachers’ summa- 
tive scores from students’ statewide test data remains somewhat unclear to district officials. 

Teachers view themselves as responsible for their own professional growth and are somewhat 
skeptical of school and district professional development 

District officials in all five study districts said that teachers who are not on a remediation 
plan 3 have some autonomy in setting their professional learning goals, although they tend 


Two districts apply 
a value-added 
model to statewide 
student test data 
and share the 
resulting teacher 
scores with 
principals during 
the summer; the 
principals in turn 
discuss the scores 
with teachers 
in the fall 
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Table 1. Decisions reportedly based on teacher evaluation data, by district 


District District District District District 

Decision A B C D E 


Influences on professional development for teachers 

Observation data during year ✓ ✓ ✓ ✓ ✓ 

Benchmark student assessment data during year ✓ ✓ ✓ ✓ ✓ 

Final teacher score derived from student test data 
Talent management decisions 

Identifying teacher leaders ✓ ✓ 

Assigning teachers to schools 

Assigning teachers to classrooms ✓ ✓ 

Designating teachers for remediation ✓ ✓ ✓ ✓ ✓ 

Allocating state performance pay to teachers ✓ ✓ ✓ ✓ 


Note: A check mark indicates that two or more sources within the district (central office administrators, a 
majority of responding principals, or a majority of responding teachers) reported that individual teacher evalu- 
ation data are used as a basis for the decisions listed. Additional details on districts’ evaluation data-based 
decision processes are in appendix C. 

Source: Authors' analysis of data from interviews with central office administrators and from surveys of princi- 
pals and teachers provided by the Arizona Department of Education, 2015. 


to collaborate with principals and instructional coaches. The particulars may vary, but at 
least 60 percent of responding principals in four districts (A, B, D, and E) reported that 
they directly oversee the professional development of the teachers they evaluate. In all five 
districts fewer than 35 percent of responding principals agreed that teachers are responsi- 
ble for designing their own professional development. 

Teachers’ perspective on responsibility for determining professional development differed 
from that of principals. At least 57 percent of responding teachers in all five districts 
reported that they are primarily responsible for using their evaluation data to guide their 
own professional development. In only two districts did a majority of teachers agree that 
their supervising administrator uses evaluation data to guide their professional growth in a 
helpful way (51 percent in district A and 53 percent in district E). 

Surveys also suggested that teachers questioned the relevance of school and district pro- 
fessional development offerings. In all five districts fewer than 50 percent of responding 
teachers agreed that administrators use evaluation data to determine the professional 
development offerings at their schools or that their district effectively links profession- 
al development offerings with the needs identified in teacher evaluations. In contrast, a 
majority of the responding principals in four districts (A, B, C, and D) reported that they 
plan their school’s professional development based on teacher evaluation data. According 
to the district officials interviewed, teachers tend not to be involved in planning profes- 
sional development at the school or district level. 

Evaluation data are not systematically used to identify teacher leaders or to assign teachers to 
schools or classrooms, but such data serve as the basis for decisions on remediation and allocation 
of state performance pay funds 

Districts A, C, and E do not have a formal process for using teacher evaluation data to 
identify teacher leaders (see table 1). However, district B uses evaluation data to identify 


In only two 
districts did 
a majority of 
teachers agree that 
their supervising 
administrator 
uses evaluation 
data to guide 
their professional 
growth in a 
helpful way 
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qualified teachers to lead professional learning communities or professional development 
opportunities at their school and at other schools in the district. In district D the superin- 
tendent reviews evaluation data when choosing an instructional coach from the available 
teacher applicants. 


Although the interviewed officials in all districts maintained that they examine the dis- 
tribution of their teachers across performance classifications — noting, for example, the 
proportion of teaching staff rated effective or above — they rarely use the data to assign 
teachers to particular schools. This finding was confirmed by responding teachers and prim 
cipals in most districts. At the same time, 80 percent of responding principals in district C 
agreed that they use teacher evaluation data to assign teachers to particular classrooms, 
subjects, or grade levels within their schools, and the district D principal explained that he 
relies on evaluation data to create effective gradedevel teams. Surveys from the other study 
districts (A, B, and E) indicated that within-school assignments are not consistently based 
on evaluation data; rather they are made informally by school leaders. 

District officials and a majority of responding principals in all five study districts main- 
tained that evaluation data are used to identify teachers for remediation (and subsequent 
improvement planning). Individual teachers’ evaluation data also influence the allocation 
of state performance pay funds, which is mandated by Arizona’s Proposition 301 (box 3): 
teachers with higher performance classifications receive additional funds in every district 
except district E, where Proposition 301 funds go to schools that meet benchmarks select- 
ed by the district’s teachers. 


Officials in 
all districts 
maintained that 
although they 
examine the 
distribution of their 
teachers across 
performance 
classifications, 
they rarely use 
the data to assign 
teachers to 
particular schools 


Teachers were more skeptical than administrators about the benefits of their new evaluations 


The new teacher evaluation process has improved teachers’ instructional practice, accord- 
ing to responding principals in districts A (53 percent), B (70 percent), and D (where the 
principal was interviewed rather than surveyed) and responding teachers in districts A 
(52 percent), D (58 percent), and E (60 percent). 

However, in all five districts only 13-46 percent of responding teachers agreed that the 
new evaluation process has benefitted students. In contrast, most responding principals in 
four districts (A, 65 percent; B, 78 percent; D, 60 percent; and E, 60 percent) agreed that 


Box 3. Proposition 301: Performance pay for teachers in Arizona 

In November 2000 Arizona voters passed Proposition 301, which increased the state’s sales 
tax from 5 percent to 5.6 percent to provide additional funds for education programs. Districts 
receive funds on a per pupil basis, and 40 percent of funds must be used for performance 
pay for teachers through locally established distribution details. Districts in turn developed 
performance pay plans with input from board members, administrators, teachers, principals, 
and parents. In 2012 Arizona House Bill 2823 amended state law to require that, beginning in 
the 2014/15 school year, teachers’ performance classifications (determined by Arizona Frame- 
work for Measuring Educator Effectiveness parameters) “shall account for thirty-three per cent 
of the forty per cent allocation” for performance pay (Arizona Revised Statutes §15-977). For 
details about the local distribution of Proposition 301 funds, see http://www.aztreasury.gov/ 
local-govt/revenue-distributions/prop-301/. 
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students have benefitted from the new evaluations. District officials — who designed and 
lead the systems — also offered positive views of student benefits. They emphasized that 
teachers are helped by more specific, standards-based feedback about their practice from 
principals and coaches, delivered more often during the year, and that students benefit 
from their teachers’ continuous review of their own practice as well as from teachers’ struc- 
tured review of benchmark assessment data with coaches or their fellow teachers. (Review- 
ing student assessment data is a key component of each study district’s observational 
framework.) 

At the same time, district officials acknowledged some anxiety among teachers about the 
new evaluation process, particularly related to the use of student test scores and student 
and parent surveys, which teachers suspect may be unreliable and unduly influenced by 
factors beyond their control . 4 

Implications of the study findings 


The study districts emphasize the review of data during the school year and have built 
organizational structures to support such work — for example, standards-based observation 
frameworks, benchmark assessments, professional learning communities, and instructional 
coaching and feedback. Although some structures were established independent of the 
state’s teacher evaluation regulations, they seem to support the regulations’ goal of creating 
a culture where data drive instructional decisions. 

After multiple years implementing standards-based observation frameworks and rubrics, 
district officials emphasized the importance of having common definitions and expecta- 
tions around effective teaching. Other districts seeking to increase the benefits of evalu- 
ation and create conditions for meaningful conversations about practice might consider 
working toward common definitions with language that can be used consistently by teach- 
ers, coaches, and principals, supported by menus of professional development opportunities 
specifically aligned with those definitions. 

Such shifts imply a potential new role for principals, who have traditionally allowed 
teachers autonomy over their classrooms. The new evaluation systems require principals 
to objectively rate teachers against standards, give them specific feedback (to be incorpo- 
rated alongside feedback from instructional coaches), and use the data to make strategic 
decisions. 

Principals tended to perceive their district’s new evaluations favorably, but teachers were 
more skeptical. While many teachers see a link between their evaluations and their pro- 
fessional development, they tend to feel independently responsible to seek out professional 
development opportunities that appropriately address the issues identified in their evalu- 
ations. This suggests that it may be worthwhile for school and district officials to target 
school-based professional development more closely to teachers’ needs identified through 
the evaluation system, provide additional support for teachers to find appropriate outside 
professional development, and consistently communicate their vision of the role that eval- 
uations can play in improving collaboration, feedback, and support within schools. 

Some issues raised in this study might be examined in more depth in future research. For 
example, researchers might seek to gain a more representative view of, or delve more deeply 


Districts seeking 
to increase 
the benefits of 
evaluation and 
create conditions 
for meaningful 
conversations 
about practice 
might consider 
working toward 
common definitions 
with language 
that can be used 
consistently by 
teachers, coaches, 
and principals, 
supported 
by menus of 
professional 
development 
opportunities 
specifically 
aligned with 
those definitions 
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into, teachers’ perspectives on evaluation. Alternatively, a sample of evaluated teachers 
could be followed over time to track the specific steps they take to improve their practice. 
Another study might assess principals’ preparation to support and guide teachers’ profes- 
sional development. Arizona education leaders might also be interested in more broadly 
surveying the local practices used across the state to evaluate the contributions of teachers 
to their students’ academic progress. 

Limitations of the study 


This study was limited by the selection of districts and interview participants and by survey 
response rates. Since districts (and principals and teachers within districts) volunteered to 
participate rather than being randomly selected, the results cannot be generalized beyond 
the participating districts. The survey response rate among principals was adequate, but 
the response rate among teachers was low overall (35 percent) and uneven across districts 
(below 50 percent in districts A, B, and E; see table A2 in appendix A for details). Teacher 
responses may not reflect the perceptions of the majority of teachers and may be biased; 
responses should thus be taken as no more than suggestive. Results might have differed had 
other teachers in the study districts or teachers from other districts responded. Moreover, 
given the self-reported nature of the study data, findings may be limited by respondents’ 
inaccurate recall or reluctance to address difficult or sensitive topics. 

The study did not attempt to characterize the fidelity or quality of the implementation of 
local teacher evaluations in relation to the requirements of the Arizona Framework for 
Measuring Educator Effectiveness or factors known to influence implementation fidelity. 
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Appendix A. Data sources and methodology 


To address the research questions, the study team analyzed interview transcripts and survey 
data compiled by the West Comprehensive Center on behalf of the Arizona Department 
of Education. Regional Educational Laboratory (REL) West did not directly collect any 
data for this study. This appendix describes the study’s data sources and general methods. 

Interview transcripts 

Data collection. The Arizona Department of Education adopted a purposeful stratified 
sampling approach to invite districts to participate in the study in October 2014, initially 
targeting 17 districts that varied in enrollment size, student demographics, standardized 
test performance, and teacher effectiveness measures implemented. Among the 17 districts 
invited, districts A-E volunteered to participate. The department then asked the superin- 
tendents of each district to nominate district officials for interviews who were most knowl- 
edgeable about the procedural steps and systems involved with collecting, storing, sharing, 
and using the data from the local teacher evaluation process. Between January 2015 and 
May 2015 teams of two trained field researchers from West Comprehensive Center visited 
each study district to conduct semistructured interviews with the district officials on 
behalf of the Arizona Department of Education, using an established interview protocol 
developed collaboratively by the department and REL West (see appendix B). One field 
researcher facilitated the interview, and the other recorded the responses. The interviews 
were recorded and transcribed. Follow-up phone interviews were conducted with district 
officials who were unavailable during the site visit. Nineteen district officials were inter- 
viewed (table Al). 

Analysis. While reviewing one interview transcript using an initial coding scheme derived 
from the interview protocol (see appendix B), a study team member inductively developed 
additional codes to capture pertinent details of the data use process (Miles, Huberman, 
& Saldana, 2014). That study team member then coached a second study team member 


Table Al. Roles of interview subjects, by district (n = 19) 


1 District 

Role of interview subject [ 

A 

Director of Human Resources 

Director of Research and Accountability 

Professional Development Coordinator 

B 

• Superintendent 

Director of Research and Assessment 

Educational Services/Teacher Training Administrator 

C 

• Superintendent 

Director of Assessment and Data 

Academic Coach (2) 

D 

• Superintendent 

Principal 

Data Specialist (2) 

E 

Associate Superintendent for School Improvement 

Executive Director of Human Resources 

Associate Superintendent for Curriculum and Instruction 

Instructional Coach (2) 

Source: Authors' compilation. 
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on the coding scheme, and together they reviewed the first coded transcript, establish- 
ing decision rules and basic examples for each code. Next, the two study team members 
independently coded a second transcript. A third study team member not involved in 
the initial coding discussions then assessed inter-rater reliability by dividing the coded 
transcript into 15 sections of text, comparing the first two researchers’ codes to calculate 
inter-rater reliability within each section (with reliability defined as the number of codes in 
agreement divided by the total number of codes assigned), and then averaging the within- 
section reliability across all 15 sections. Overall rater reliability was .71. The three study 
team members then met to review the three transcript sections where reliability was below 
.50. (The main discrepant codes fell within the Actor and Influence code groups, described 
in appendix B.) They reviewed the discrepant codes in these three sections and agreed on 
a set of consensus codes for the section. Some coding language was modified based on this 
discussion. The study team’s coding was then considered calibrated, and the first two study 
team members separately analyzed and coded the remaining interview transcripts using 
the final coding scheme (see appendix B). 

Online surveys of teachers and principals 

Data collection. The Arizona Department of Education (in collaboration with West Com- 
prehensive Center and REL West) developed online end-of-year surveys to gather data on 
principals’ and teachers’ perceptions and experiences with their district’s new evaluation 
systems. Surveys included sets of closed-ended questions with Likert-style response scales 
(strongly agree, agree, neither agree nor disagree, disagree, strongly disagree) to explore 
the degree to which respondents agreed with particular statements about how evaluation 
data were used locally (with response categories later collapsed for analysis; see tables Cl- 
C5). Introductory text and links to the surveys were provided to research officials in each 
study district in May 2015 for distribution to 62 principals and 2,791 teachers. Response 
rates varied by district, but the overall response rate for principals was 76 percent, and the 
overall response rate for teachers was 35 percent (table A2). 

Analysis. Analysis of the teacher and principal survey data supplemented the interview 
analysis, triangulating the qualitative information — in-depth, contextually based informa- 
tion on the local uses of teacher evaluation data — with survey information from a larger 
sample of individuals working at different levels in the school system. The quantitative 


Table A2. Survey response rates across the study districts 


District 


Principals 



Teachers 


Number of 
surveys 
distributed 

Number of 
respondents 

Response 

rate 

(percent) 

Number of 
surveys 
distributed 

Number of 
respondents 

Response 

rate 

(percent) 

A 

23 

17 

74 

1,309 

510 

39 

B 

12 

10 

83 

656 

130 

20 

C 

10 

5 

50 

325 

183 

56 

D 

na 

na 

na 

41 

26 

63 

E 

17 

15 

88 

460 

122 

27 

Total 

62 

47 

76 

2,791 

971 

35 


na is not applicable because the principal in district D was interviewed rather than surveyed. 
Source: Authors' analysis of data provided by the Arizona Department of Education, 2015. 
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survey results thus complement the qualitative results. For the survey item analysis the 
study team generated and tabulated the response frequencies for each survey item for each 
participating district. 

To further explore the teacher survey results, the study team also examined whether, for 
each district, responding high school teachers or responding teachers with more experience 
— in particular, those at their current school for 5 years or more as well as those with more 
than 10 years of overall teaching experience — perceived these issues differently, that is, 
whether a significantly ( p < .05) larger or smaller proportion agreed with each survey item. 
Any such differences are noted in the case summaries in appendix C. 

Deriving case summaries and findings 

The two study team members who coded the interviews organized the transcript data 
by codes (see appendix B), which enabled them to identify patterns and themes for the 
case summaries (complemented by the survey data), which in turn allowed the tracking 
of common themes across districts. The full study team relied on themes prevalent within 
districts (themes raised by district administrators were compared with the relevant survey 
results from principals or teachers) and then across districts. Cross-case themes were drawn 
from the case summaries and included as study findings if the theme was raised (and sub- 
stantiated) in three or more of the districts. As noted, each district was treated as an inde- 
pendent case because of the local variations in system design and implementation. 
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Appendix B. Protocol and code book for district interviews 


This appendix presents the interview protocol used to gather data from district officials, 
as well as the code book used to analyze and summarize the data from the interview tran- 
scripts (table Bl). 

Interview questions for district officials 
Background 

1. I’d first like to get an idea of your background: 

a. What is your current role in this district? 

Probe: Inquire about job title and specific responsibilities. 

b. How long have you worked in this position? 

c. What previous positions have you held in this district and elsewhere? 

d. In what ways are you involved with the district’s teacher evaluation process? 

Probe: Inquire about specific responsibilities, particularly related to results data. 

Uses of teacher evaluation data 

Next I’d like to learn more about how your district collects, stores, shares, and makes use of 
the results from teacher evaluations. 

2. We understand that your district uses multiple measures to evaluate teacher effective- 
ness. How are results from the different measures (that is, each of the different teacher 
ratings or scores) collected and stored? What specific steps are involved? 

Probe: For example, are teachers’ observation ratings written on paper forms? Entered 
into a spreadsheet? Entered into an online database? Is any video reviewed? If so, how 
is it rated or evaluated? Who enters the data, the teacher’s supervising administrator? 

3. Once the teacher evaluation data are collected, who is allowed to access the results? 
How are results of teacher evaluation data collection shared? With whom? 

Probe: Do principals have access to a central data system that allows them to analyze 
teacher effectiveness data? How do they use it? 

4. What guidance does the district provide to principals (if any) about how to use teacher 
effectiveness data/scores to make decisions related to teacher hiring, assignment, pro- 
fessional development, or contract renewal or dismissal? 

5. To what extent, and in what ways, are a teacher’s evaluation results used to promote 
his or her professional growth? How does this process work? 

a. Do principals recommend or require specific next steps for teachers? If so, can you 
provide some examples? How do they decide what to require? 

b. If teachers are responsible for their own professional growth, what kinds of activi- 
ties do they engage in to grow? How do they decide what to do? 


B-l 


c. What specific professional development (PD) options are available to teachers? 
Probe: How much of the PD offerings are provided in-person? Online? How do 
principals deploy master or peer teachers, department heads, or instructional lead- 
ership team members to coach, mentor, and model best practices for teachers? How 
does this work, specifically? 

d. To what extent are PD resources or support activities directly aligned with the 
specific indicators on the district’s instructional framework or observation rubric? 

e. To what extent do principals track teachers’ participation in PD opportunities? 
How does this happen? 

6. Are evaluation results used to identify potential teacher leaders and/or grant highly 
effective teachers additional leadership responsibilities, for example, as mentors or peer 
coaches? 

Probe: If YES, how does this process work? If NO, why not? 

7. Are evaluation results used to assign teachers to particular schools or classrooms? For 
example, are highly effective teachers assigned to teach students most in need of effec- 
tive instruction? 

Probe: If YES, how does this process work? If NO, why not? 

8. How are evaluation results used to determine improvement plans for teachers identi- 
fied as needing remediation? 

9. How might a teacher’s evaluation results influence the decision to retain (or offer 
tenure to) that teacher? 

10. To what extent is teacher compensation linked to evaluation results? How does this 
work? How has this been received? 

11. From your perspective, what results from certain measures are more useful than others. . . 

a. For teachers? 

b. For school and/or district administrators? 

12. From your perspective, how has your district’s new teacher evaluation process led to 
changes in instructional practices? How have these changes affected learning out- 
comes for students in your district? 

a. What unique contextual factors or conditions have enabled effective data use in 
your district? Has this been helpful? Why or why not? 

b. Alternatively, what factors or conditions constrained your district from using eval- 
uation data in beneficial ways? Can you elaborate? 

13. From your perspective, how has the district’s new teacher evaluation process influenced 
the professional environment or climate in your district’s schools? 

Probe: For example, do professionals across the school (or district) now think and 
talk differently about instruction? Are any longstanding norms, values, or procedures 
changing? 
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Final thoughts 


14. Are teacher evaluation results being used in your district in other ways we haven’t 
discussed? 

Probe: For example, perhaps to evaluate the effectiveness of certain trainings or PD 
programs? 

15. Are there any district reports or system documents that you can share with us that 
describe how your district is using evaluation data to improve teaching? 

16. Do you have any other final thoughts or comments that you would like to share? Was 
there anything else I should have asked about? 


Table Bl. Code book for interview transcripts 

Code 

Description 

Actor: Role group using evaluation data 

District administrator 

All central office staff 

School administrator 

Principals and assistant principals 

Teacher 

All current classroom teachers 

Types of data collected, stored, or used for decisionmaking 

Evaluation reports 

Reports that contain multiple forms of evaluation data collected, stored, or used 

Summative ranking 

Teachers' final performance classification (that is, highly effective, effective, developing, or 
ineffective) 

Observation data 

Data on teachers' practice collected via an instructional rubric 

Student achievement data 

Student academic progress data, including raw student achievement data as well as value-added 
measures or student learning objectives 

Teacher evaluation rubrics 

Research-based instructional rubrics that define high-quality instruction in different domains 
(that is, classroom environment or assessment) and are used to make decisions on professional 
development offerings for teachers 

Other 

Other types of data that may or may not be included in the teacher evaluation process, including 
student demographic data, surveys from students or teachers, or teachers peer assessments 

Data collection process 

Collect and store 

Describing the collection or storage process for evaluation data 

Create measures 

Measures or scores created from raw observation or student achievement data 

Sharing 

Describing the act of sharing the data with principals, teachers, or other district staff 

Access data 

Noting that certain actors may have direct access to data while others may not 

Enter data 

Referencing the entry of different data elements into specific databases 

Compile data 

Creating value-added measures or disaggregating data (in graphs or reports) to make them 
available for use; usually done by a central office research or assessment official 

Training: Data collection 

Training provided to various actors on how to collect and store evaluation data for later use 

Training: Data use 

Training provided to administrators on how to understand and use the different types of 
evaluation data in decisionmaking 

Format: District created database 

Database created specifically to collect and store evaluation data 

Format: Existing database 

Existing database used to collect and store evaluation data 

Format: Other 

Data stored in other (nondatabase) formats 

(continued) 
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Table Bl. Code book for interview transcripts (continued) 

Code 

Description 

Data use for professional development 

Plan created: District 

Districtwide professional development opportunities created by central office administrators 

based on teachers’ instructional needs 

Plan created: Whole school 

Schoolwide professional development plans or schoolwide areas of focus created based on 

teachers’ instructional needs 

Plan created: Individual 

Individual professional development plan for teachers based on teachers’ areas of need; 
different from a support plan designed to remediate teachers at risk based on their ineffective 
rankings 

Delivery format 

The various professional development options or formats from which principals, coaches, and 

teachers can choose 

Oversight: Creation of plan 

An actor’s responsibility to use evaluation data to create a professional 
development plan 

Oversight: Execution of plan 

An actor’s responsibility to monitor the execution of a professional development plan 

Data use for talent management 

Assignment 

Actor uses evaluation data to assign teachers to specific grade levels, classrooms, or schools 
based on evaluation data 

Retention 

Actor uses evaluation data to decide whether or not to retain teachers 

Remediation 

Actor uses evaluation data to place ineffective teachers on an improvement plan to improve their 
instructional practice 

Compensation 

Actor uses evaluation data to compensate teachers 

Identification of teacher leaders 

Actor uses evaluation data to identify teacher leaders or mentor teachers at the district or school 

level 

Oversight 

The actor responsible for making talent management decisions 

Influences on data use and on the outcomes of data use 

Data reliability 

Reference to actor not considering certain data as reliable or accurate and influencing the extent 
to which data are used 

Resources or tools 

The tools or resources that actors may consider and influence their data use 

Support or capacity 

Actor considering people’s skill for using or understanding data (which may be supported by 
training) as influencing data use 

Time 

Any mention of time in regards to the use of evaluation data 

Help 

Factors in the evaluation system considered to support data use or outcomes 

Hindrance 

Factors in the evaluation system considered to hinder data use or outcomes 

Perceived outcomes of data use 

Teaching quality 

Reference to whether or how teachers improved their instructional practice based on the use of 
evaluation data for professional development 

Culture or climate 

Reference to whether or how the culture or climate surrounding data use within the district or its 
schools may have shifted due to the new evaluation system 

Source: Authors' compilation. 
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Appendix C. Case summaries 


This appendix presents case summaries for the five study districts. Each case summary 
describes the local processes for collecting, accessing, sharing, and using teacher evalua- 
tion data, discusses key implications, and concludes with a summary table of principal and 
teacher survey results from the district. 

District A 

District A was the largest study district, enrolling more than 25,000 students in 2013/14 
(more than 50 percent of whom were a racial/ethnic minority) and received a letter grade 
of A from the state. 

Survey and interview sample. Information was collected from a structured group inter- 
view with the district’s director of human resources, director of research and accountabili- 
ty, and professional development coordinator (see table A1 in appendix A), as well as from 
surveys of 17 of the district’s 23 principals (74 percent response rate) and of 510 of its 1,309 
teachers (39 percent response rate; see table A2 in appendix A). 

Eleven responding principals led an elementary or grade K-8 school, five led a high school, 
and one led a grade K-12 school. Although respondents had a range of overall experience 
as principals, 47 percent had been at their current school for two years or less. 

Of the 510 responding teachers, 37 percent reported teaching primarily in an elementary 
school, 16 percent reported teaching primarily in a grade K-8 school, 15 percent reported 
teaching primarily in a middle school, and 31 percent reported teaching primarily in a 
high school. These proportions generally align with the district A teacher population, of 
which 72 percent teach in a grade K-8 school and 28 percent teach in a high school, 
according to administrative data obtained from the district. 

Responding teachers were generally experienced teachers: 58 percent reported having 
more than 10 years of experience, 27 percent reported having 5-10 years of experience, 
and 14 percent reported having fewer than 5 years of experience. Experience among the 
district’s full teacher population was more balanced: 42 percent had more than 10 years of 
experience, 33 percent had 5-10 years of experience, and 25 percent had fewer than 5 years 
of experience, according to district data. In surveys the perceptions of district A teachers 
with more than 10 years of experience did not differ significantly from the perceptions 
of teachers with less experience on any question. Responding teachers had also been at 
their current school for varying amounts of time: 38 percent reported having been at their 
current school for 2 years or less, 17 percent reported having been at their current school 
for 3-4 years, 31 percent reported having been at their current school for 5-10 years, and 
13 percent reported having been at their current school for more than 10 years. 

Responding teachers also reported working in a variety of areas: elementary grades 
(34 percent), math (13 percent), special education (12 percent), reading/language arts 
(11 percent), social studies (7 percent), science (5 percent), career and technical education 
(5 percent), the arts (4 percent), physical education (4 percent), and various other assign- 
ments (5 percent). 
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Collecting, accessing, and sharing teacher evaluation data. District A evaluates teachers 
using data collected from classroom observations of teachers and from students’ statewide 
standardized test scores, applying an evaluation framework designed for use in 2012/13 (to 
align with the Arizona Framework for Measuring Educator Effectiveness) by a commit- 
tee of teachers, principals, and district curriculum and instruction specialists and research 
staff. Principals and teachers have access to teachers’ observation and student growth data 
and final performance classifications through an internal online data system created and 
maintained by the district’s research and accountability department. The system aggre- 
gates teachers’ observation and value-added scores to create final performance classifica- 
tions and reports. 

Over the school year teachers are observed multiple times by their supervising administra- 
tor (using the Danielson Framework for Teaching, which includes four domains — planning 
and preparation, classroom environment, instruction, and professional responsibilities — 
each with five to six subcomponents), who enters data into the district database. According 
to district officials, all principals and assistant principals have been trained and certified to 
collect observation data and score observations consistently and accurately. 

The system emails teachers the data from their most recent observation, and they can view 
the data online. Each teacher also has a student academic progress score, which is calcu- 
lated during the summer through a value-added model using data from statewide standard- 
ized tests in reading, math, and science (for teachers in grades or content areas that are 
subject to statewide testing) or based on test results aggregated to the grade or school level 
(for teachers in grades or content areas that are not subject to statewide testing). Teachers 
receive their score as part of the evaluation report containing their final performance clas- 
sification for the school year. These summative teacher reports can also be accessed online 
by district officials and school administrators. 

Surveys indicate that the evaluation data collection process is clear to principals and teach- 
ers. All responding principals and 90 percent of responding teachers reported knowing 
what types of information are collected during teacher evaluations, and 73 percent of 
teachers reported knowing how evaluation data will be used by their administrators (table 
Cl). This knowledge may relate to longevity at the school — 79 percent of responding 
teachers who reported having been at their current school for five years or more reported 
knowing how the data will be used, compared with 68 percent of teachers who had been at 
their school for fewer than five years ( p = .02). 

Using teacher evaluation data to shape professional development. Participating district 
and school administrators said teachers’ evaluation data shape their individual profession- 
al development plans, while schoolwide and districtwide professional development plans 
are based on overarching areas of need across teachers and schools. The district provides 
professional development opportunities in multiple formats, such as face-to-face coaching 
or online videos, aligned with the definitions of high-quality instruction in the Daniel- 
son Framework for Teaching, in areas that site leaders at multiple schools have identified 
as needing improvement. School leaders seem to appreciate this connection more than 
teachers — 53 percent of responding principals (but 24 percent of responding teachers) 
agreed that district A does a good job linking its professional development offerings with 
the needs identified through teachers’ evaluations (see table Cl). 
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Table Cl. District A principals’ and teachers’ agreement with statements about their experiences with 

the district’s teacher evaluation process (percent) 






Principals (n = 

17) 


Teachers ( n = 

510) 



Neither 

Disagree 



Neither 

Disagree 


Agree or 

agree 

or 


Agree or 

agree 

or 


strongly 

nor 

strongly 


strongly 

nor 

strongly 

Statement 

agree 

disagree 

disagree 

Statement 

agree 

disagree 

disagree 

1 know what types of information 




1 know what types of information 




are collected during the teacher 

100.0 

0.0 

0.0 

are collected during the teacher 

90.2 

4.3 

5.5 

evaluation process in my district. 




evaluation process in my district. 




1 have access to a central data 




1 know how data collected during 




system that allows me to analyze 
my teachers’ evaluation data/ 
effectiveness scores. 

100.0 

0.0 

0.0 

my evaluation will be used by 

school administrators. 

72.7 

13.9 

13.4 

After their evaluations, teachers at 




After my evaluation, the next steps 




my school clearly understand the 
next steps they need to take for 
their professional growth. 

88.2 

11.8 

0.0 

1 need to take for my professional 
growth are clear to me. 

69.8 

15.8 

14.4 

The teachers 1 evaluate have 




1 engage in professional 




the opportunity to engage in 




development opportunities directly 




professional development 

94.1 

0.0 

5.9 

linked to the needs identified in my 

54.6 

20.9 

24.5 

opportunities directly linked to the 

needs identified in their evaluations. 




evaluation. 




1 directly oversee the professional 




My supervising administrator uses 




development of the teachers 1 

94.1 

5.9 

0.0 

evaluation results to guide my 

51.2 

23.3 

25.4 

evaluate. 




professional growth in a helpful way. 




Teachers in my school are primarily 




I'm primarily responsible for using 




responsible for overseeing their 

31.3 

25.0 

43.8 

evaluation results to guide my own 

78.5 

15.2 

6.3 

own professional development. 




professional development. 




1 use the data collected during 




Administrators use the data 




teacher evaluations to determine 

the professional development 

94.1 

5.9 

0.0 

collected during teacher evaluations 
to determine the professional 

29.5 

32.1 

38.4 

offerings at my school. 




development offerings at my school. 




From my perspective, the district 




From my perspective, the district 




does a good job linking its 




does a good job linking its 




professional development offerings 

52.9 

17.7 

29.4 

professional development offerings 

24.4 

31.5 

44.2 

with the needs identified through 




with the needs identified in 




teachers’ evaluations. 




teachers’ evaluations. 




In my district, teacher evaluation 




District administrators use teacher 




results are used to assign teachers 

5.9 

11.7 

82.4 

evaluation results to assign 

7.9 

48.3 

43.7 

to particular schools. 




teachers to particular schools. 




1 use teacher evaluation results 




School administrators use teacher 




to assign teachers to particular 

47.1 

17.7 

35.3 

evaluation results to assign 

21.1 

44.6 

34.3 

classrooms, subjects, or grade 

teachers to particular classrooms, 

levels. 




subjects, or grade levels. 




Poor-performing teachers in 




Poor-performing teachers in 




my school are designated for 

remediation based on their 

70.6 

23.5 

5.9 

my district are designated for 

remediation based on their 

38.6 

38.4 

23.0 

evaluation results. 




evaluation results. 




My district provides me with 




In my district, evaluation results 




adequate guidance about how 




are used to determine which 




to use teacher evaluation data/ 
effectiveness scores to make 

decisions. 

70.6 

17.7 

11.8 

teachers are retained. 

32.6 

39.7 

27.7 

(continued) 
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Table Cl. District A principals’ and teachers’ agreement with statements about their experiences with 

the district’s teacher evaluation process (percent) (continued) 





Principals (n = 

17) 


Teachers ( n = 

510) 



Neither 

Disagree 



Neither 

Disagree 


Agree or 

agree 

or 


Agree or 

agree 

or 


strongly 

nor 

strongly 


strongly 

nor 

strongly 

Statement 

agree 

disagree 

disagree 

Statement 

agree 

disagree 

disagree 

In my school, evaluation results are 
used to promote teachers. 

5.9 

29.4 

64.7 

In my district, evaluation results 
are used to promote teachers. 

12.1 

36.2 

51.7 

In my school, a teacher's 




In my district, a teacher's 




evaluation results impacts the 
amount of compensation (pay) he 

11.8 

35.3 

52.9 

evaluation results impacts the 
amount of compensation (pay) he 

14.4 

23.0 

62.6 

or she receives. 




or she receives. 




From my perspective, the 




The teacher evaluation process 




district's new teacher evaluation 
process has improved teachers’ 
instructional practice. 

52.9 

41.2 

5.9 

used in my district has led me to 
improve my instructional practice. 

51.6 

21.7 

26.7 

From my perspective, the district’s 




The teacher evaluation process 




new teacher evaluation process 

64.7 

29.4 

5.9 

used in my district has benefited 

43.7 

26.3 

30.0 

has benefited students. 




my students. 




Note: Percentages may not sum to 100 across rows because of rounding. 




Source: Authors' analysis of data provided by the Arizona Department of Education, 2015. 





Within schools, principals have the autonomy to set schoolwide professional development 
plans, which tend to be informed by evaluation data, delivered during the time set aside 
by the district for whole-school professional development, and selected from the district’s 
menu of professional development options, district officials noted. In surveys, 94 percent of 
principals agreed that they use the data collected during teacher evaluations to determine 
the professional development offerings at their school. Here again, however, a smaller pro- 
portion of teachers (30 percent) agreed that this was the case (see table Cl). 

Teachers in district A have the authority to establish, in collaboration with their super- 
vising administrator, their own professional development plans (except for teachers who 
receive a final performance classification of ineffective and in turn are provided with spe- 
cific supports via a formal improvement plan). In formal preconferences at the beginning 
of the school year, principals and teachers review prior evaluation data to set learning 
goals for the year. Teachers can select the types of professional development activities they 
attend, based on their learning goals. Indeed, 79 percent of responding district A teachers 
reported that they were primarily responsible for using evaluation data to guide their own 
professional development. 

Throughout the school year, teachers and their supervisor monitor and discuss their prog- 
ress, ultimately evaluating accomplishments at a formal end-of-year conference (where 
they begin to set goals for the following year). In surveys, 88 percent of responding district 
A principals agreed that, after their evaluations, teachers at their school clearly under- 
stand the next steps they need to take for their professional growth. Teachers endorsed this 
sentiment as well, with 70 percent agreeing that the next steps they need to take for their 
professional growth are clear to them after their evaluations (see table Cl). 

Using teacher evaluation data for talent management. Although professional develop- 
ment opportunities appear to be well linked to the needs identified in teachers’ evaluations 
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in district A, evidence suggests that use of these data to strategically assign teachers or to 
identify teacher leaders is informal and remains a work in progress. Forty'Seven percent of 
responding principals agreed that they use evaluation data to assign teachers to particular 
classrooms, subjects, or grade levels within their school (35 percent disagreed). Evidence 
also suggests that evaluation data are not being used to assign teachers to particular schools 
in district A. District administrators suggested that using data to make within-school 
teacher assignments is an emerging process, one that district officials are seeking to for- 
malize. In addition, while district administrators suggest that district A publicly encour- 
ages teacher leadership and that schools use mentor teachers to coach new or struggling 
teachers, there is no evidence to suggest that the district or principals systematically use 
evaluation data to identify teachers for such leadership opportunities. 

In district A, evaluation data influence decisions such as remediation that are more tradi- 
tionally associated with the evaluation process. According to district officials, any district 
A teacher who receives one or more unsatisfactory ratings on the Danielson Framework 
for Teaching’s 22 components or any teacher with four or more years of experience who 
receives four or more basic ratings is given a final performance classification of ineffective 
and is provided specific supports via a formal improvement plan. In surveys, 71 percent of 
principals agreed that poor-performing teachers in district A are designated for remedia- 
tion based on their evaluation data (see table Cl). Interviewed district A officials explained 
that — beginning with the 2014/15 school year, as required by state law — teachers at differ- 
ent performance classifications would receive different stipends from Proposition 301 funds 
(as designated by the district’s teacher evaluation working group). 

Implications of teacher evaluation data use. District officials maintained that the dis- 
trict’s extensive training and multiple years of using the Danielson Framework for Teach- 
ing have embedded the rubric’s language and definitions into the everyday work of teachers 
and principals and ensured the consistent and reliable rating of instruction. At the same 
time, district officials noted that, although generally confident in the reliability of their 
observations, some teachers have expressed concerns about the derivation of their final 
performance classifications, given the limited measures available. In surveys, 53 percent 
of principals and 52 percent of teachers agreed that district A’s teacher evaluation process 
has improved instructional practice, and 65 percent of principals agreed that it has ben- 
efited students. However, a smaller proportion of teachers (44 percent) agreed that the 
process has benefitted students (see table Cl). A smaller proportion of district A’s high 
school teachers agreed that the teacher evaluation process has benefited their students 
(38 percent, compared with 46 percent of non-high school teachers; p = .01). 

District B 

District B enrolled more than 10,000 students in 2013/14 (approximately 30 percent of 
whom were a racial/ethnic minority) and received a letter grade of A from the state. 

Survey and interview sample. Information was collected from a structured group inter- 
view with the district’s superintendent, director of research and assessment, and educa- 
tional services/teacher training administrator (see table A1 in appendix A), as well as from 
surveys of 10 of the district’s 12 principals (83 percent response rate) and of 130 of its 656 
teachers (20 percent response rate; see table A2 in appendix A). 
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Six of the responding district B principals led an elementary school, and four led a middle 
or high school. Although respondents had a range of overall experience as principals, 
60 percent had been at their current school for two years or less. 

Of the 130 responding teachers, 52 percent reported teaching primarily in an elementary 
school, 18 percent reported teaching primarily in a middle school, 25 percent reported 
teaching primarily in a high school, and 5 percent reported teaching across grade spans. 
These proportions generally align with the district B teacher population. 

Responding teachers were generally experienced teachers: 47 percent reported having 
more than 10 years of experience, 34 percent reported having 5-10 years of experience, 
and 19 percent reported having fewer than 5 years of experience. Across the full district 
B population of teachers, 39 percent reported having more than 10 years of experience, 
33 percent reported having 5-10 years of experience, and 28 percent reported having fewer 
than 5 years of experience. Responding teachers had also been at their current school for 
varying amounts of time: 42 percent reported having been at their current school for 2 
years or less, 21 percent reported having been at their current school for 3-4 years, and 
38 percent reported having been at their current school for 5 or more years. 

Responding district B teachers also reported working in a variety of areas: elementary 
grades (37 percent), special education (16 percent), reading/language arts (14 percent), 
science (9 percent), math (8 percent), the arts (8 percent), social studies (4 percent), and 
world languages (3 percent). 

Collecting, accessing, and sharing teacher evaluation data. District B was an early 
adopter of evaluation reform, according to the superintendent. The district has used the 
Strange Teacher Effectiveness Performance Evaluation System (and its associated online 
data system) for teacher evaluation since 2011 (box Cl). 

District B teachers can always access their updated evaluation data (and associated learn- 
ing plans) in the district’s online data system, district officials reported. The information 
is reviewed during teachers’ end-of-year conferences with their supervising administrator 
and again during their preconference at the beginning of the next school year. Principals 
receive their teachers’ student progress scores via email during the summer. According to 
district officials, principals also receive whole-school reports from the district displaying 
how their teachers performed on each standard that year, which the principals can use to 
identify patterns (for example, by grade level or content area) to determine the school’s key 
professional development needs. 

Surveys suggested common understanding of the evaluation data collected — all princi- 
pals and 81 percent of teachers reported knowing what types of information are collected 
during teacher evaluations, and 65 percent of teachers agreed that they know how their 
evaluation data will be used by their administrators (table C2). This understanding varied 
by grade span taught: 45 percent of responding high school teachers agreed, compared 
with 72 percent of responding non-high school teachers, a statistically significant differ- 
ence ( p = .01). 

Using teacher evaluation data to shape professional development. After reviewing 
teachers’ evaluation data during the summer and identifying areas of need, principals 
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Box Cl. The Stronge Teacher Effectiveness Performance Evaluation System 


The Stronge Teacher Effectiveness Performance Evaluation System assesses teachers on seven 
standards, six of which are observed and rated by school administrators via multiple informal and 
formal classroom visits during the year (with data and comments tracked online) and one of which 
(student progress) is calculated separately by the district’s research and assessment division by 
applying the district’s value-added model to student achievement data from Arizona’s Instrument 
to Measure Standards (AIMS) or the ACT college readiness exam for high school students. 

The seven standards assessed are professional knowledge, instructional planning, instruc- 
tional delivery, assessment, learning environment, professionalism, and student progress. 
Teachers are rated ineffective, partially effective, effective, or highly effective on each stan- 
dard. To create an overall summative performance classification that aligns with the require- 
ments of the Arizona Framework for Measuring Educator Effectiveness, teachers’ ratings for 
the student progress standard are weighted at 34 percent, and their ratings for the other six 
standards are each weighted at 11 percent. The district is piloting the use of the Stronge sys- 
tem’s student surveys to allow teachers to adjust to the data before data are incorporated into 
their learning environment ratings. 

Teachers in grades and content areas that are subject to statewide standardized tests 
receive a student progress rating based on data from AIMS or the ACT, and teachers of sub- 
jects whose students do not take standardized tests can choose the exam (AIMS/ACT) and 
content area (for example, math or reading) that will be used to aggregate their students’ 
performance for their student progress rating. 


Table C2. District B principals’ and teachers’ agreement with statements about their experiences with 

the district’s teacher evaluation process (percent) 
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Table C2. District B principals’ and teachers’ agreement with statements about their experiences with 

the district’s teacher evaluation process (percent) (continued) 
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10.0 
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77.8 
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37.2 
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37.2 
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my students. 




Note: Percentages may not sum to 100 across rows because of rounding. 




Source: Authors' analysis of data provided by the Arizona Department of Education, 2015. 
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communicate the needs to the district’s instructional coaches, district officials explained. 
Funded by Proposition 301, district B’s coaches provide support around curriculum and 
instruction but do not play a formal role in teachers’ evaluation. After summer planning, 
the principal may elect to have coaches work with teachers online through the district’s 
integrated system, during afterschool sessions, or via one-on-one coaching. Participation 
in district B’s professional development courses is tracked online. District administrators 
emphasized the importance of having teachers’ professional development and evaluation 
data linked in an online system, both to make the process seamless for teachers and to 
allow for principal oversight. While teachers have some discretion in their individual pro- 
fessional development choices, the training reportedly has to be aligned with their evalua- 
tion data, according to district officials. 

In surveys, district B teachers were more skeptical than principals about whether their 
school and district connected evaluation data with professional development, and teachers 
tended to view themselves — rather than administrators — as the primary agents of their 
learning and growth. Seventy-one percent of teachers reported that they are primarily 
responsible for using evaluation data to guide their own professional development, and 
only 44 percent agreed that their supervising administrator uses evaluation data to guide 
their professional growth in a helpful way. Principals tended to acknowledge the district’s 
efforts in this area, with 70 percent of responding principals agreeing that district B does a 
good job linking its professional development offerings with the needs identified in teach- 
ers’ evaluations (see table C2). However, 20 percent of responding teachers agreed with the 
same statement. 

Fifty-nine percent of responding district B teachers agreed that the next steps they need to 
take for their professional growth were clear to them after their evaluations, and 46 percent 
reported that they engaged in professional development opportunities linked directly to 
the needs identified in their evaluations (see table C2). 

More skepticism was evident among high school teachers and among teachers who have 
been at their school for five years or more: a significantly smaller proportion of high school 
teachers (than non-high school teachers) and teachers who have been at their school for 
five years or more (than those with fewer years at the site) agreed that their supervising 
administrator uses evaluation data to guide their professional growth in a helpful way and 
that the district does a good job linking its professional development offerings with the 
needs identified in teachers’ evaluations ( p < .05). 

Using teacher evaluation data for talent management. Participating district B offi- 
cials said they train principals to make talent management decisions, and 90 percent of 
responding district principals agreed that the district provides them with adequate guid- 
ance about how to use teacher evaluation data to make decisions. District B teachers who 
do not meet their learning targets and are rated low on other Stronge standards are placed 
on improvement plans, which include substantial intervention in their areas of weakness, 
district officials reported. Evaluation data are also used to identify high-performing teach- 
ers, who are in turn called on to lead their schools’ professional learning communities and 
to lead professional development sessions at their school and other schools in the district. 
Moreover, according to district officials, teachers who meet certain student achievement 
benchmarks are provided stipends using Proposition 301 funds. 
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Decisions regarding teacher assignments to schools are handled at the central office, among 
the superintendent, human resources staff, and principals, as well as at the school level, 
among principals and gradedevel teams. While the district provides data to principals and 
supports their decisionmaking, district officials emphasized that principals make final deck 
sions regarding placements in their schools. At the same time, principal and teacher survey 
data indicate that it is not a common practice in district B to use teacher evaluation data 
to assign teachers to schools or classrooms (see table C2). 

Implications of teacher evaluation data use. Participating district officials emphasized 
that they have worked hard in recent years to instill a culture rooted in collaboration and 
data-based decisionmaking, for example, by regularly dedicating time for collaboration, 
data review, and feedback, by having teachers assume leadership roles, by having leaders 
reference supporting data and evidence in all their decisions, and by hiring people with 
the experience and desire to work in a data-centric workplace. This culture has helped 
facilitate the implementation of teacher evaluations, which are seen as aligning with 
much of the work that was already underway in district B prior to adoption of the Arizona 
Framework for Measuring Educator Effectiveness. 

At the same time, district officials acknowledged that teachers are apprehensive about being 
evaluated based on student test data and student survey data, as such measures may be influ- 
enced by outside factors unrelated to their teaching effectiveness. Perhaps partly because of 
this apprehension, district B teachers were more dubious of their evaluation’s impact than 
were principals, who on surveys tended to agree that the district’s new teacher evaluation 
process has improved teachers’ instructional practice (70 percent agreed) and benefited stu- 
dents (78 percent agreed). A smaller proportion of teachers agreed that their evaluations 
improved teaching (47 percent) and benefited students (37 percent; see table C2). 

District C 

District C enrolled more than 6,000 students in 2013/14 (more than 50 percent of whom 
were a racial/ethnic minority) and received a letter grade of B from the state. 

Survey and interview sample. Information was collected from structured interviews with 
the district’s superintendent, director of assessment and data, and two academic coaches 
(see table A1 in appendix A), as well as from surveys of 5 of the district’s 10 principals 
(50 percent response rate) and of 183 of its 325 teachers (56 percent response rate; see table 
A2 in appendix A). 

Of the 183 responding teachers, 57 percent reported teaching primarily in an elementary 
school, 24 percent reported teaching primarily in a middle school, and 19 percent reported 
teaching primarily in a high school. These proportions generally align with the district C 
teacher population. 

The survey drew disproportionately from district C’s most experienced teachers: 43 percent 
of responding teachers reported having more than 10 years of experience, 26 percent report- 
ed having 5-10 years of experience, and 31 percent reported having fewer than 5 years of 
experience. According to administrative data obtained from the district, among the dis- 
trict’s full teacher population, 4 percent had more than 10 years of experience, 26 percent 
had 5-10 years of experience, and 70 percent had fewer than 5 years of experience. Survey 


C-10 


respondents reported less experience at their current school: 36 percent reported having 
been at their current school for 2 years or less, and 32 percent reported having been at 
their current school for 3-4 years. 

Responding teachers also reported working in a variety of areas: elementary grades 
(39 percent), special education (15 percent), math (11 percent), reading/language arts 
(10 percent), the arts (8 percent), physical education (4 percent), science (3 percent), social 
studies (3 percent), career and technical education (3 percent), and various other assign- 
ments (3 percent). 

Collecting, accessing, and sharing teacher evaluation data. District C has evaluated 
its teachers using the Arizona Department of Education’s teacher evaluation model since 
August 2012, when it began participating in the state’s voluntary pilot. The state model 
assesses teachers using classroom observations by principals during the year (based on the 
Danielson Framework for Teaching); end-of-year online surveys of students, parents, and 
peer teachers; and measures of student academic progress based on teaching assignment, as 
defined by state-created rating tables. A portion of all district C teachers’ student academic 
progress rating is determined by their performance on student learning objectives focused 
on students’ growth over the course of the year, as determined by district assessment data 
gathered via a pretest, three benchmark tests, and a posttest. 

Once the various evaluation data have been collected from the relevant measures — 
observation data, survey data, prior-year state test scores, and current-year benchmark 
assessment data — a state-trained district data official enters the data into the state-designed 
spreadsheet appropriate for the teaching assignment, and embedded formulas calculate the 
teacher’s summative performance classification (see http://www.azed.gov/teacherprincipal 
-evaluation/teacher-rating-tables/ for the state-developed spreadsheets and rating tables). 

Participating district C officials noted that their teachers and principals are generally well 
versed in the Danielson Framework for Teaching because the district adopted it several 
years prior to the state’s 2011 evaluation regulations. Site administrators preconference 
with teachers before formal observations to discuss the lesson and its goals and then record 
their ratings and notes during the observation (either by hand or via laptop or tablet). 
Data are then shared with the teacher during a post-conference and reviewed during the 
end-of-year evaluation conference. To maintain common understanding and expectations 
and promote inter-rater reliability, the district hosts several video-based calibration and 
training sessions for principals during the year. Teachers can also regularly access their 
students’ benchmark assessment data online via the district’s data system, and district C 
instructional coaches reported that they review these data in their weekly meetings with 
teachers. In surveys, 73 percent of district C teachers agreed that they know what types of 
information are collected during the evaluation process. 

Using teacher evaluation data to shape professional development. Evidence suggests that 
professional development offerings are not formally aligned with teacher evaluation data 
in district C. In surveys, 40 percent of responding principals and 16 percent of responding 
teachers agreed that the district does a good job linking its professional development offer- 
ings with the needs identified through teachers’ evaluations, and 38 percent of respond- 
ing teachers agreed that after their evaluation, the next steps they need to take for their 
professional growth are clear to them (table C3). Responding high school teachers were 
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Table C3. District C principals’ and teachers’ agreement with statements about their experiences with 

the district’s teacher evaluation process (percent) 
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Table C3. District C principals’ and teachers’ agreement with statements about their experiences with 

the district’s teacher evaluation process (percent) (continued) 
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evaluation results impacts the 
amount of compensation (pay) he 

100.0 

0.0 

0.0 

evaluation results impacts the 
amount of compensation (pay) he 

70.7 

16.0 

13.3 

or she receives. 




or she receives. 




From my perspective, the 




The teacher evaluation process 




district's new teacher evaluation 
process has improved teachers’ 
instructional practice. 

20.0 

20.0 

60.0 

used in my district has led me to 
improve my instructional practice. 

24.2 

28.0 

47.8 

From my perspective, the district’s 




The teacher evaluation process 




new teacher evaluation process 

20.0 

20.0 

60.0 

used in my district has benefited 

12.6 

21.9 

65.6 

has benefited students. 




my students. 




Note: Percentages may not sum to 100 across rows because of rounding. 




Source: Authors' analysis of data provided by the Arizona Department of Education, 2015. 





particularly skeptical on this front: a significantly smaller proportion of responding district 
C high school teachers than of non-high school teachers reported engaging in profession- 
al development opportunities linked directly to the needs identified in their evaluation 
(37 percent versus 51 percent; p = .04) and that the next steps for their professional growth 
were clear to them after their evaluations (18 percent versus 43 percent; p = .02). 

To identify priority areas for professional development and to design the annual schedule 
of districtwide learning opportunities, the district’s professional development coordina- 
tor gathers feedback from principals and coaches during regular meetings as well as from 
teachers (through an online survey), according to the district officials interviewed. In addi- 
tion to employing six instructional coaches, district C holds four early-release professional 
development days each year, during which teachers can attend various workshops. 

Principals can monitor their teachers’ participation in these activities through attendance 
records, and teachers must submit a professional development portfolio under domain 
4 of the Danielson Framework for Teaching (professional responsibilities). However, in 
surveys 20 percent of responding principals agreed that they directly oversee the profes- 
sional development of the teachers they evaluate, and 45 percent of teachers agreed that 
their supervising administrator uses evaluation data to guide their professional growth 
in a helpful way (see table C3). Moreover, 59 percent of responding teachers reported 
that they were primarily responsible for using evaluation data to guide their own profes- 
sional development, whereas 20 percent of responding principals indicated that this was 
the case. 

Using teacher evaluation data for talent management. District C trains principals to 
make talent management decisions during the district’s annual summer leadership retreat. 
In surveys, 60 percent of principals agreed that the district provides them with adequate 
guidance about how to use teacher evaluation data to make decisions. For example, teach- 
ers rated as ineffective must complete an improvement plan, and the principal and instruc- 
tional coach work closely with the teacher to identify specific action items and timelines. 
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Meanwhile, teachers rated effective or highly effective are provided with a bonus from 
Proposition 301 funds (as determined by a committee of teachers and administrators). 
Evaluation data are not used to identify teacher leaders or to assign teachers to particular 
schools; instead, such assignments are based on teacher preferences and school needs (as 
determined by district and school leaders). However, 80 percent of responding principals 
reported that they use teacher evaluation data to assign teachers to particular classrooms, 
subjects, or grade levels — though 12 percent of responding teachers agreed that this was 
the case (see table C3). 

Implications of teacher evaluation data use. Participating district officials maintained 
that the new accountability imposed by the evaluation system has created a culture that 
values quality instruction and continuous improvement, and noted that having used the 
Danielson Framework for Teaching for several years has raised expectations and facilitat- 
ed valuable dialogue about instructional strategies between principals and teachers and 
among teachers during professional learning community meetings. District officials also 
noted that teachers appear to be embracing the use of data to improve their work. Instruc- 
tional coaches — who are not part of the formal evaluation process — now meet more reg- 
ularly with teachers to review data and noted improvements in instruction as a result of 
their work with new and struggling teachers. 

However, district officials admitted that newer evaluation measures — in particular, student 
and parent surveys and student learning objectives (measured through a series of bench- 
mark tests during the year) — have confused and worried teachers. Indeed, responding 
teachers tended to be skeptical of the district’s evaluation process. Less than 25 percent of 
responding principals and teachers reported that the district’s teacher evaluation process 
has improved teachers’ instructional practice or benefited students (see table C3). While 
they claimed that understanding is growing among principals and teachers, district offi- 
cials acknowledged that ongoing communication, training, and support will be vital to the 
system’s success. 

District D 

District D enrolled fewer than 1,000 students in 2013/14 (more than 85 percent of whom 
were a racial/ethnic minority) and received a letter grade of C from the state. 

Survey and interview sample. Information was collected from structured interviews with 
the district’s superintendent, school principal, and two data specialists (see table A1 in 
appendix A), as well as from surveys of 26 the district’s 41 teachers (63 percent response 
rate; see table A2 in appendix A). 

Respondents were generally experienced teachers: 75 percent reported having been at their 
current school for at least 5 years, and 73 percent reported having more than 10 years 
of experience. The proportion of teachers with more than 10 years of experience aligns 
with the district D teacher population, 77 percent of whom had more than 10 years of 
experience. 

Of the responding teachers, 46 percent reported teaching elementary school, and others 
reported teaching reading/language arts, math, science, special education, social studies, 
and physical education (with 8-12 percent in each subject). 
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Collecting, accessing, and sharing teacher evaluation data. District D also adopted the 
Arizona Department of Education’s teacher evaluation model in 2012, and since then has 
evaluated teachers using that set of multiple measures: observations by principals based on 
the Danielson Framework for Teaching; end-of-year online surveys of students, parents, 
and peer teachers; and measures of student academic progress based on teaching assign- 
ment, as defined by state-created rating tables (and including at least two student learning 
objectives based on data from district benchmark assessments). 

The district D principal records his ratings and notes from classroom observations in 
one online data system, and students’ pre- and post-test scores that serve as the basis for 
teachers’ student learning objectives are stored online in another database. Teachers can 
access both databases to review their observation data and student achievement data. The 
student, parent, and peer survey data are collected via a separate online survey tool that is 
accessible only to district data officials. 

Once the teacher’s relevant evaluation data are collected and entered into the appropri- 
ate state-designed spreadsheet, embedded formulas calculate the teacher’s summative per- 
formance classification. A summary of the data for each teacher is then provided to the 
district D principal for discussion during the teacher’s end-of-year evaluation conference. 
Surveys suggest that teachers tend to understand the process: 89 percent reported knowing 
what types of information are collected during teacher evaluations, and 58 percent report- 
ed knowing how data collected during their evaluation will be used by school administra- 
tors (table C4). 

Using teacher evaluation data to shape professional development. In interviews, district 
and school leaders in district D maintained that observation and student academic prog- 
ress data from the evaluation process are being used to design districtwide, schoolwide, and 
individual teacher’s professional development plans. They did not indicate whether or how 
parent and student survey data were being used. District officials emphasized that the district’s 
sustained implementation of the Danielson Framework for Teaching since 2012 has helped 
focus the district’s efforts to align professional development offerings with the framework. 

The superintendent explained that the district’s review of the observed levels of teacher 
practice across its schools helped target district trainings toward particular areas — for 
example, creating effective classroom relationships and environments. The superintendent 
added that the district also eliminated in-school suspensions based on evaluation data 
regarding teachers’ relationships with students. Similarly, the principal pointed out that 
he reviews observation data with the district’s instructional coaches to define teachers’ key 
growth areas and target schoolwide improvements, such as in differentiated instruction 
and student engagement. Schoolwide professional development is offered every Friday for 
two hours at the school through in-person trainings or professional learning communities. 
However, responding district D teachers seemed less convinced of the connection between 
their evaluation data and the professional development offered by the school and district. 
Twenty- three percent agreed that the district does a good job linking its professional devel- 
opment offerings with the needs identified in teachers’ evaluations, and 39 percent agreed 
that administrators use evaluation data to determine the professional development offer- 
ings at their school (see table C4). This perception varied by experience level: 32 percent 
of responding teachers with more than 10 years of experience agreed, and no teachers with 
less experience agreed ( p = .02). 
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Table C4. District D teachers’ agreement with statements about their experiences with the district’s 
teacher evaluation process (percent) 

Statement 

Agree or 
strongly agree 

Teachers (n = 26) 

Neither agree 
nor disagree 

Disagree or 
strongly disagree 

1 know what types of information are collected during the teacher 
evaluation process in my district. 

88.5 

3.9 

7.7 

1 know how data collected during my evaluation will be used by school 

administrators. 

57.7 

30.8 

11.5 

After my evaluation, the next steps 1 need to take for my professional 
growth are clear to me. 

57.7 

26.9 

15.4 

1 engage in professional development opportunities directly linked to the 
needs identified in my evaluation. 

53.9 

30.8 

15.4 

My supervising administrator uses evaluation results to guide my 
professional growth in a helpful way. 

46.2 

26.9 

26.9 

I’m primarily responsible for using evaluation results to guide my own 
professional development. 

57.7 

42.3 

0.0 

Administrators use the data collected during teacher evaluations to 
determine the professional development offerings at my school. 

38.5 

50.0 

11.5 

From my perspective, the district does a good job linking its professional 
development offerings with the needs identified in teachers’ evaluations. 

23.1 

46.2 

30.8 

District administrators use teacher evaluation results to assign teachers 
to particular schools. 

7.7 

76.9 

15.4 

School administrators use teacher evaluation results to assign teachers 
to particular classrooms, subjects, or grade levels. 

50.0 

30.8 

19.2 

Poor-performing teachers in my district are designated for remediation 

based on their evaluation results. 

15.4 

76.9 

7.7 

In my district, evaluation results are used to determine which teachers 

are retained. 

23.1 

61.5 

15.4 

In my district, evaluation results are used to promote teachers. 

3.9 

50.0 

46.1 

In my district, a teacher's evaluation results impacts the amount of 
compensation (pay) he or she receives. 

52.0 

16.0 

32.0 

The teacher evaluation process used in my district has led me to 
improve my instructional practice. 

57.7 

26.9 

15.4 

The teacher evaluation process used in my district has benefited my students. 

42.3 

38.5 

19.2 


Note: Percentages may not sum to 100 across rows because of rounding. Because district D has only one principal, only teachers were 
surveyed, and the principal was interviewed. 

Source: Authors' analysis of data provided by the Arizona Department of Education, 2015. 


To support the professional growth of individual teachers, district and school leaders reported 
that the principal and instructional coaches collaborate with teachers to help them develop 
their individual professional learning plans based on their evaluation data. To do so, the 
principal reviews evaluation data and suggests various ways for the teacher to improve his 
or her practice, such as accessing instructional videos or meeting one-on-one with a par- 
ticular instructional coach. He also explained that each grade-level grouping has instruc- 
tional coaches to help teachers analyze their observation data or their students’ benchmark 
assessment data to plan new instructional strategies and consider next steps for professional 
development (some of the instructional coaches were hired specifically to facilitate student 
data-driven discussions with teachers). These connections are supported by teacher survey 
data — 58 percent of district D teachers agreed that the next steps they need to take for their 
professional growth were clear to them after their evaluations, and 54 percent agreed that 
they engage in professional development opportunities linked directly to the needs identified 
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in their evaluations (see table C4). At the same time, 58 percent agreed that they are primar- 
ily responsible for using evaluation data to guide their own professional development. 

Using teacher evaluation data for talent management. Evidence suggests that district 
and school leaders in district D are using teacher evaluation data to make strategic talent 
management decisions, not only regarding remediation, retention, and compensation — 
teachers rated effective or highly effective receive $500 stipends from Proposition 301 
funds (a plan developed by teachers, according to the superintendent) — but also to identi- 
fy teacher leaders and assign teachers to particular classrooms. For example, when recently 
considering teachers for a special assignment instructional coaching position, the super- 
intendent examined the observation and student assessment data for effective and highly 
effective applicants only. The principal also relies on evaluation data when shifting teach- 
ers to try to create effective grade-level teams. 

Implications of teacher evaluation data use. According to the superintendent and the 
principal, district D’s evaluation system has helped focus discussions between the principal 
and teachers and among teachers on the effective practices defined in the Danielson Frame- 
work for Teaching as well as on student progress data. The principal said that the evalua- 
tion system’s focus on multiple observations and reviewing student data has allowed teachers 
to get the whole picture of their instruction, which has made them reflective practitioners 
and benefitted students. And district data officials added that teachers find student learning 
objectives helpful because the process helps them focus on monitoring student progress and 
differentiating instruction. The superintendent concurred, noting that teachers are now dis- 
aggregating and analyzing individual students’ data to plan next steps, a recent occurrence. 
There appears to be consensus that the evaluation process has improved teaching practice in 
district D. Agreeing with their district and school leaders, 58 percent of responding teachers 
reported that the system has led them to improve their instructional practice (see table C4). 

However, some district officials expressed concerns that some teachers are merely com- 
plying with the evaluation system’s rules rather than authentically embracing the reflec- 
tion and continuous improvement aspects of the process. Forty-two percent of responding 
teachers agreed that district D’s teacher evaluation process has benefited their students, 
and 19 percent disagreed (see table C4). 

District E 

District E enrolled more than 9,000 students in 2013/14 (more than 70 percent of whom 
were a racial/ethnic minority) and received a letter grade of B from the state. 

Survey and interview sample. Information was collected from interviews with the dis- 
trict’s associate superintendent for school improvement, executive director of human 
resources, associate superintendent for curriculum and instruction, and two instructional 
coaches (see table A1 in appendix A), as well as from surveys of 15 of the district’s 17 prin- 
cipals (88 percent response rate) and of 122 of its 460 teachers (27 percent response rate; 
see table A2 in appendix A). 

Ten responding principals led an elementary school, and five led a middle school. Respond- 
ing principals had a range of experience, both at their current school and overall as princi- 
pals: a third of respondents reported more than 10 years of experience as a principal, and a 
third reported having been an administrator at their current school for more than 10 years. 
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TwO'thirds of responding teachers reported teaching primarily in an elementary school, 
and one-third reported teaching primarily in a middle school. These proportions gen- 
erally align with the district’s teacher population. Responding teachers were general- 
ly experienced teachers: 59 percent reported having more than 10 years of experience, 
23 percent reported having 5-10 years of experience, and 18 percent reported having fewer 
than 5 years of experience. According to administrative data obtained from the district, 
41 percent of the district’s full teacher population had more than 10 years of experience in 
2014/15, 24 percent had 5-10 years of experience, and 35 percent had fewer than 5 years of 
experience. Responding teachers had been at their current school for varying amounts of 
time: 31 percent reported having been at their current school for 2 years or less, 32 percent 
reported having been at their current school for 3-10 years, 37 percent reported having 
been at their current school for more than 10 years. 

Responding teachers also reported working in a variety of areas: elementary school 
(51 percent), special education (16 percent), reading/language arts (11 percent), math 
(6 percent), social studies (5 percent), physical education (4 percent), science (4 percent), 
and the arts (3 percent). 

Collecting, accessing, and sharing teacher evaluation data. Teacher evaluations in 
district E are based on observations of teachers using the Marzano Teacher Evaluation 
Model — which assesses teachers across four domains: classroom strategies and behaviors 
(with 41 rated elements), preparing and planning (8 elements), reflecting on teaching (5 
elements), and collegiality and professionalism (6 elements) — as well as on school-level 
student achievement growth data from district benchmark assessments. District E switched 
from the Danielson Framework for Teaching to the Marzano Teacher Evaluation Model in 
2012 . 

Principals or assistant principals conduct multiple formal and informal observations of 
teachers’ classroom practice throughout the school year, often immediately entering data 
into the district’s online data system, which compiles the ratings and feedback principals 
give teachers on the different elements of the Marzano Teacher Evaluation Model. Coaches 
and peer teachers also informally observe district E teachers and provide formative instruc- 
tional feedback directly to the teacher, but this information is not shared with administra- 
tors and does not contribute to the teacher’s final performance classification. Teachers can 
access the online system and review this information over time. 

In addition to observation data, the district incorporates student growth data from district 
benchmark assessments into teachers’ final performance classifications. District E students 
take these local benchmark assessments four times during the school year, and school 
leaders and teachers can access the district’s online testing system to review their students’ 
data. At the end of the school year, the student academic progress portion of teachers’ per- 
formance classifications are awarded based on the aggregate growth of the students across 
the entire school. District E officials reported that they are still trying to determine the 
best way to incorporate student test data into teacher evaluations. 

Surveys suggest that the data collection process is generally understood by participants 
— all responding principals and 85 percent of responding teachers agreed that they know 
what types of information are collected during teacher evaluations, and 65 percent of 
responding district E teachers agreed that they know how the data collected during their 
evaluation will be used by school administrators (table C5). 
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Table C5. District E principals’ and teachers 

.’ agreement with statements based on their experiences 

with the district’s teacher evaluation process (percent) 





Principals (n = 

15) 


Teachers (n = 

122) 



Neither 

Disagree 



Neither 

Disagree 


Agree or 

agree 

or 


Agree or 

agree 

or 


strongly 

nor 

strongly 


strongly 

nor 

strongly 

Statement 

agree 

disagree 

disagree 

Statement 

agree 

disagree 

disagree 

1 know what types of information 




1 know what types of information 




are collected during the teacher 

100.0 

0.0 

0.0 

are collected during the teacher 

85.1 

9.1 

5.8 

evaluation process in my district. 




evaluation process in my district. 




1 have access to a central data 




1 know how data collected during 




system that allows me to analyze 
my teachers’ evaluation data/ 
effectiveness scores. 

100.0 

0.0 

0.0 

my evaluation will be used by 

school administrators. 

64.8 

20.5 

14.8 

After their evaluations, teachers at 




After my evaluation, the next steps 




my school clearly understand the 
next steps they need to take for 
their professional growth. 

53.3 

40.0 

6.7 

1 need to take for my professional 
growth are clear to me. 

66.4 

23.0 

10.7 

The teachers 1 evaluate have 




1 engage in professional 




the opportunity to engage in 




development opportunities directly 




professional development 

66.7 

33.3 

0.0 

linked to the needs identified in my 

59.5 

25.6 

14.9 

opportunities directly linked to the 

needs identified in their evaluations. 




evaluation. 




1 directly oversee the professional 




My supervising administrator uses 




development of the teachers 1 

60.0 

26.7 

13.3 

evaluation results to guide my 

52.5 

29.5 

18.0 

evaluate. 




professional growth in a helpful way. 




Teachers in my school are primarily 




I’m primarily responsible for using 




responsible for overseeing their 

13.3 

33.3 

53.3 

evaluation results to guide my own 

78.5 

15.7 

5.8 

own professional development. 




professional development. 




1 use the data collected during 




Administrators use the data 




teacher evaluations to determine 

the professional development 

42.9 

35.7 

21.4 

collected during teacher evaluations 
to determine the professional 

43.3 

34.2 

22.5 

offerings at my school. 




development offerings at my school. 




From my perspective, the district 




From my perspective, the district 




does a good job linking its 




does a good job linking its 




professional development offerings 

40.0 

33.3 

26.7 

professional development offerings 

40.2 

29.5 

30.3 

with the needs identified through 




with the needs identified in 




teachers’ evaluations. 




teachers’ evaluations. 




In my district, teacher evaluation 




District administrators use teacher 




results are used to assign teachers 

13.3 

0.0 

86.7 

evaluation results to assign 

11.5 

49.2 

39.3 

to particular schools. 




teachers to particular schools. 




1 use teacher evaluation results 




School administrators use teacher 




to assign teachers to particular 

26.7 

20.0 

53.3 

evaluation results to assign 

20.5 

45.1 

34.4 

classrooms, subjects, or grade 

teachers to particular classrooms, 

levels. 




subjects, or grade levels. 




Poor-performing teachers in 




Poor-performing teachers in 




my school are designated for 

remediation based on their 

93.3 

6.7 

0.0 

my district are designated for 

remediation based on their 

45.1 

39.3 

15.6 

evaluation results. 




evaluation results. 




My district provides me with 




In my district, evaluation results 




adequate guidance about how 




are used to determine which 




to use teacher evaluation data/ 
effectiveness scores to make 

decisions. 

53.3 

40.0 

6.7 

teachers are retained. 

41.3 

41.3 

17.4 

In my school, evaluation results are 
used to promote teachers. 

7.1 

14.3 

78.6 

In my district, evaluation results 
are used to promote teachers. 

15.6 

43.4 

41.0 

(continued) 
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Table C5. District E principals’ and teachers 

.’ agreement with statements based on their experiences 

with the district’s teacher evaluation process (percent) (continued) 





Principals (n = 

15) 


Teachers (n = 

122) 



Neither 

Disagree 



Neither 

Disagree 


Agree or 

agree 

or 


Agree or 

agree 

or 


strongly 

nor 

strongly 


strongly 

nor 

strongly 

Statement 

agree 

disagree 

disagree 

Statement 

agree 

disagree 

disagree 

In my school, a teacher's 




In my district, a teacher's 




evaluation results impacts the 
amount of compensation (pay) he 

26.7 

13.3 

60.0 

evaluation results impacts the 
amount of compensation (pay) he 

38.0 

22.3 

39.7 

or she receives. 




or she receives. 




From my perspective, the 




The teacher evaluation process 




district's new teacher evaluation 
process has improved teachers’ 
instructional practice. 

46.7 

26.7 

26.7 

used in my district has led me to 
improve my instructional practice. 

59.5 

22.3 

18.2 

From my perspective, the district’s 




The teacher evaluation process 




new teacher evaluation process 

60.0 

33.3 

6.7 

used in my district has benefited 

45.9 

27.9 

26.2 

has benefited students. 




my students. 




Note: Percentages may not sum to 100 across rows because of rounding. 




Source: Authors' analysis of data provided by the Arizona Department of Education, 2015. 





Principals and district administrators regularly review teachers’ performance classifications 
across the district to see whether schools have similar distributions of teachers ranked 
highly effective, effective, developing, and ineffective, according to one district official. 

Using teacher evaluation data to shape professional development. District officials 
reported in interviews that they are working toward aligning districtwide professional 
development sessions (which are delivered five times each year) with the definitions of 
high-quality instruction outlined in the Marzano Teacher Evaluation Model. Each district 
E school also has regular early release days devoted to whole-school professional develop- 
ment, commonly involving presentations and small group discussions. Teachers may also 
choose to attend afterschool professional development sessions provided by instructional 
coaches, and can access the various Marzano resources as needed. Surveys indicated that 
this aligned support remains a work in progress. Forty percent of responding principals 
and 40 percent of responding teachers agreed that the district does a good job linking its 
professional development offerings with the needs identified through teachers’ evaluations 
(see table C5). 

Principals have individual conversations with teachers at the beginning of the school year 
and devise an improvement plan based on their evaluation data from the previous year, 
according to the instructional coaches interviewed. All district E teachers develop indi- 
vidual professional growth plans, not just those who rated low on Marzano-based obser- 
vations. Throughout the school year, principals observe teachers and give them feedback 
using the online observation system. In addition, instructional coaches help teachers 
analyze their students’ benchmark test data during weekly team meetings, and work indi- 
vidually with teachers to ensure that they devise actions plans to change their instruction 
based on these data. Responding district E teachers tended to agree that the link between 
their evaluations and their professional development was strong — 66 percent reported that 
the next steps they need to take for their professional growth were clear after their evalu- 
ations, and 60 percent agreed that they engage in professional development opportunities 
directly linked to the needs identified in their evaluations. And 53 percent agreed that 
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their supervising administrator uses evaluation data to guide their professional growth in a 
helpful way. Each of these statements was supported by a majority of responding principals 
as well (see table C5). 

Using teacher evaluation data for talent management. Evidence suggests that teacher 
evaluation data are not being used systematically to identify teacher leaders or to assign 
teachers to different schools or classrooms in district E. The district also does not dif- 
ferentiate teachers’ pay based on their evaluation data, according to school and district 
officials. Proposition 301 funds are instead dispersed to all teachers in particular schools 
according to a varied set of performance indicators determined by the district’s teachers, 
including the school’s letter grade from the state, schoolwide growth on benchmark assess- 
ments, and parent and student survey feedback. District and school leaders reported that 
evaluation data are used to identify teachers for remediation. Those on remediation plans 
due to poor performance must identify specific skill areas within the Marzano Teacher 
Evaluation Model to develop in partnership with an instructional coach. Principals then 
monitor their progress, and if they do not improve, their contracts are not renewed, district 
officials reported. 

Implications of teacher evaluation data use. District officials and instructional coaches 
maintained in interviews that, under district E’s evaluation system, teachers are receiving 
more specific and timely feedback on their practice and are more often tracking their stu- 
dents’ progress on benchmark assessments. District officials also reported that the district’s 
online observation data system has made observations more efficient, allowing principals 
to visit classrooms, observe, and give feedback more often than in previous years. The 
interviewed instructional coaches agreed and suggested that teachers are growing more 
comfortable receiving and discussing online feedback from principals, and are requesting 
more feedback, more often. Sixty percent of responding district E teachers agreed that the 
district’s evaluation process has led them to improve their instructional practice, although 
a smaller proportion of responding principals (47 percent) agreed that this was the case. 
Conversely, 60 percent of principals reported that the process has benefitted students, and 
46 percent of teachers agreed that this was the case. 
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Notes 


1. Value-added models attempt to measure a teacher’s impact on student test scores inde- 
pendent of other factors affecting achievement and use different statistical methods to 
adjust for students’ prior performance and background characteristics 

2. In district A, value-added scores are not part of the evidence used in remediation 
decisions; only teachers with one or more unsatisfactory ratings (or four or more basic 
ratings for more experienced teachers) are assigned a final performance classification of 
ineffective. In district B principals receive teachers’ value-added scores and associated 
standard errors, and teachers receive a rating of 4 (highly effective) for Stronge stan- 
dard 7 (student progress) if their value-added score is not statistically significant. 

3. A teacher is generally placed in remediation status (and required to establish a doc- 
umented plan for improvement) when he or she is rated ineffective or unsatisfactory 
during one or more evaluations. 

4. Similar suspicions were raised in teacher focus groups across the sample of districts 
studied in Ruffini et al.’s (2014) study of the implementation of new teacher evalua- 
tions in Arizona. 
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